Dynamical mean field theory for models of confluent tissues and beyond

We consider a recently proposed model to understand the rigidity transition in confluent tissues and we study its dynamical behavior under several types of dynamics: gradient descent, thermal Langevin noise and active drive. We derive the dynamical mean field theory equations and integrate them numerically and compare the results with numerical simulations. In particular we focus on gradient descent dynamics and show that this algorithm is blind to the zero temperature replica symmetry breaking (RSB) transition point. In other words, even if the Gibbs measure at zero temperature is RSB, the algorithm is able to find its way to a zero energy configuration. This is somehow expected and agrees with previous findings in numerical simulations on other examples of continuous constraint satisfaction problems. Our results can be also straightforwardly applied to the study of high-dimensional regression tasks where the fitting functions are non-linear functions of a set of weights found via the optimization of the square loss.Comment: 17 pages, 3 figures, Submission to SciPos

Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape

Stochastic Gradient Descent (SGD) is an out-of-equilibrium algorithm used extensively to train artificial neural networks. However very little is known on to what extent SGD is crucial for to the success of this technology and, in particular, how much it is effective in optimizing high-dimensional non-convex cost functions as compared to other optimization algorithms such as Gradient Descent (GD). In this work we leverage dynamical mean field theory to analyze exactly its performances in the high-dimensional limit. We consider the problem of recovering a hidden high-dimensional non-linearly encrypted signal, a prototype high-dimensional non-convex hard optimization problem. We compare the performances of SGD to GD and we show that SGD largely outperforms GD. In particular, a power law fit of the relaxation time of these algorithms shows that the recovery threshold for SGD with small batch size is smaller than the corresponding one of GD.Comment: 5 pages + appendix. 3 figure

Dynamical mean field theory for models of confluent tissues and beyond

We consider a recently proposed model to understand the rigidity transition in confluent tissues and we derive the dynamical mean field theory (DMFT) equations that describes several types of dynamics of the model in the thermodynamic limit: gradient descent, thermal Langevin noise and active drive. In particular we focus on gradient descent dynamics and we integrate numerically the corresponding DMFT equations. In this case we show that gradient descent is blind to the zero temperature replica symmetry breaking (RSB) transition point. This means that, even if the Gibbs measure in the zero temperature limit displays RSB, this algorithm is able to find its way to a zero energy configuration. We include a discussion on possible extensions of the DMFT derivation to study problems rooted in high-dimensional regression and optimization via the square loss function